Skip to content

RAG optimisation: answer quality (P1-P10), v17 numeric grounding, offline fallback, hybrid RPC hardening + golden-recall regression fix - #118

Merged
BigSimmo merged 5 commits into
mainfrom
claude/pedantic-bassi-f858e1
Jul 2, 2026
Merged

RAG optimisation: answer quality (P1-P10), v17 numeric grounding, offline fallback, hybrid RPC hardening + golden-recall regression fix#118
BigSimmo merged 5 commits into
mainfrom
claude/pedantic-bassi-f858e1

Conversation

@BigSimmo

@BigSimmo BigSimmo commented Jul 2, 2026

Copy link
Copy Markdown
Owner

Summary

  • Answer quality (P1–P10): synthesis prompt rewritten (targeting-first, clinician voice, few-shot, strict verbatim-numeric rules; cache key v13→v17); responsiveness gate extended to model answers; natural fallback wording; number-safe source truncation; unverified figures un-bolded + faithfulness caveat deduplicated; per-document crowding cap (incl. retry paths); query-class-aware strong reasoning effort + 30s answer timeout.
  • Offline/source-only fallback: RAG_PROVIDER_MODE=auto|openai|offline, embedding-free lexical retrieval, fail-closed on weak evidence, answerQualityTier labels + UI disclosure; insufficient_quota distinguished from transient rate limits.
  • Hybrid retrieval hardening: all four hybrid RPCs de-drifted/fixed (ambiguous-id + 130s plpgsql seqscan), structured RPC-failure logging/telemetry, pg_trgm typo correction, OR-relaxation fallback; 23-case golden retrieval guardrail added.
  • Regression fix for main: the recent uncapped selection score + blanket source-governance metadata weighting in retrieval-selection.ts regressed the golden eval to 16/23 (doc-recall@5 1.0→0.76) on the partially-enriched corpus. Restored relevance-first clamped ordering; governance stays enforced in ranking penalties and the answer layer. Governance selection test amended with measured justification.
  • Merged with origin/main twice (incl. PR Codify live retrieval RPCs, reclaim dead vector indexes, unify schema health #117's codified live RPCs — confirmed those snapshots carry the fixed language sql definitions).

Measured results: golden retrieval 23/23 (doc/content-recall@5 1.0, mrr 0.743) · eval:rag grounded 15/15 · eval:quality grounded-supported 0.833→0.933 · numeric-warning rate 0.175→0.125, v17 probe 6→1 unverified tokens · p95 latency 54s→17s · offline proof: 0 OpenAI tokens, grounded source-only answers, fail-closed on weak evidence.

Verification

  • npm run verify:cheap (736 passed / 2 skipped, lint + typecheck clean)
  • npm run verify:ui (UI change is a single disclosure note; covered by unit tests, Chromium run deferred to protect API budget)
  • npm run verify:release
  • npm run format:check
  • npm run check:production-readiness (READY; Supabase target confirmed)

Clinical Governance Preflight

  • Source-backed claims still require linked source verification before clinical use (numeric-faithfulness gate strengthened: unverified figures un-bolded, hard fail-closed in actionable dose/threshold context)
  • No patient-identifiable document workflow was introduced or expanded without explicit governance approval
  • Supabase target remains Clinical KB Database (sjrfecxgysukkwxsowpy)
  • Service-role keys and private document access remain server-only
  • Demo/synthetic content remains clearly separated from real clinical sources
  • Source metadata, review status, and outdated/unknown-source behavior remain conservative (governance moved out of retrieval selection ordering — where it buried correct documents on a partially-enriched corpus — and remains enforced in ranking penalties and the answer/source-governance layer; unknown ≠ bad)
  • Deployment classification/TGA SaMD impact was checked when clinical decision-support behavior changed (no new decision-support capability; answers remain source-gated and fail-closed)

Notes

🤖 Generated with Claude Code

BigSimmo and others added 5 commits July 2, 2026 10:01
…ieval hardening

Retrieval / database:
- De-drift and fix all four hybrid RPCs (ambiguous-id 42702 + plpgsql seqscan 130s regression);
  migrations for chunks/index-units/embedding-fields/memory-cards, schema-health execution smoke,
  dead-variant cleanup, and a pg_trgm clinical-term typo-correction function
- Structured hybrid-RPC failure logging + telemetry (hybrid_rpc_errors) so silent layer death
  can never hide again; golden retrieval eval expanded 10 -> 23 cases (23/23 green)
- OR-relaxation + trigram typo-correction fallbacks for over-conjunctive tsqueries (8a/8b)

Answer generation quality (P1-P10):
- Synthesis prompt rewritten: targeting-first, positive clinician voice, few-shot examples,
  strict numeric verbatim rules (no invented ranges/levels); cache key v13 -> v17
- Responsiveness gate extended to model answers on simple direct non-definition questions
- Natural generation-timeout fallback wording; offline extractive phrasing softened
- Query-class-aware strong reasoning effort (high only for dose/threshold) + 30s answer timeout;
  procedural "what is required" queries no longer false-fail the definition fragment gate
- Number-safe, sentence-boundary source truncation for model context (truncateForModel)
- Unverified figures are un-bolded and the faithfulness caveat is deduplicated (single CRITICAL
  gap); per-document crowding cap (3 chunks/doc) on model context including retry paths
- Values-only bolding policy; flattened-table run-on separation safety net

Offline / provider resilience:
- RAG_PROVIDER_MODE=auto|openai|offline with embedding-free lexical retrieval, fail-closed
  source-only answers, answerQualityTier labels, and UI source-only disclosure
- insufficient_quota distinguished from transient rate limits; auto-degrade on provider failure

Validated: verify:cheap 706 passed; retrieval golden 23/23 (recall 1.0); eval:rag 15/15 grounded;
eval:quality grounded-supported 0.833 -> 0.933, numeric warnings 0.175 -> 0.125 (v17 probe: 6 -> 1
unverified tokens on regression queries); p95 latency 54s -> ~17s; offline proof 0 OpenAI tokens.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…factor with worktree P1-P10

Resolutions:
- applyNumericVerification: combined policies — keep this branch's un-bolding of unverified
  figures and single-caveat dedupe, adopt main's hard fail-closed gate for unverified numbers
  in actionable dose/threshold/monitoring context (numeric_faithfulness_gate_source_gap)
- generateWithModel: keep options-object shape, query-class-aware strong reasoning effort, and
  P9-capped retry contexts; adopt main's broader truncation retry (strong-route retries too);
  drop main's now-dead modelTierUsed in favour of usedStrongModel/retriedWithStrong flags
- sourceBackedGenerationTimeoutAnswer: keep this branch's natural source-pointer wording
  (validated against the source-inventory quality detector)
- env/.env.example: keep documented non-pro strong tier + 30s answer timeout; adopt main's
  runtime pro-model guard (runtimeAnswerModel) via requestedOpenAIAnswerModels import
- tests: rag-answer-fallback withhold assertion made bold-marker tolerant; content-accuracy
  tests updated for the merged two-tier numeric policy (caveat path + hard-gate path)

verify:cheap green after merge: 736 passed / 2 skipped.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ecall 0.76 -> 1.0

The merged main change to retrieval-selection.ts (uncapped candidate score + blanket source-
governance metadata weighting) regressed the 23-case golden retrieval eval to 16/23
(doc-recall@5 1.0 -> 0.76, mrr 0.75 -> 0.64). Root cause, measured by bisection:
- live hybrid/finalScore values saturate at 1.0, so uncapping let boost stacking fully override
  lexical relevance ordering within the saturated region
- the corpus is only partially metadata-enriched; normalizeSourceMetadata coerces unenriched
  documents to unknown/unverified, so blanket status penalties (-0.21) vs enriched boosts (+0.14)
  swung ranking by ~0.35 for reasons unrelated to relevance and buried correct documents

Fix: restore the clamp on the candidate relevance score and remove governance metadata weighting
from selection ordering entirely (intent boosts retained). Governance stays enforced by ranking
penalties and the answer/source-governance layer; RC8 (source-strength filtering) remains tracked
in docs/rag-hybrid-findings-and-todo.md. Amended the retrieval-selection governance test to the
corrected contract with the measured justification.

Validated: golden retrieval 23/23 (doc-recall@5 1.0, content-recall@5 1.0, top-k 1.0, mrr 0.743);
retrieval-selection tests 7/7; verify:cheap 736 passed / 2 skipped; eval:rag 5/5 grounded.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…dentally resurrected

DashboardFloatingFab.tsx, ui/button.tsx, ui/icon-button.tsx, ui/skeleton.tsx, and lib/tools.ts
were deleted on main (component extraction/cleanup); a git checkout during retrieval-regression
bisection restored the old copies into the worktree and they were committed unintentionally.
No current source imports them.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@BigSimmo
BigSimmo merged commit f24cd2c into main Jul 2, 2026
2 checks passed
@BigSimmo
BigSimmo deleted the claude/pedantic-bassi-f858e1 branch July 2, 2026 08:12
BigSimmo added a commit that referenced this pull request Jul 2, 2026
…ata debt at 0.6 (#176)

* test(eval): scope danger-warning failures to answered routes; re-accept metadata debt

Two release-quality-eval governance fixes, no app-behavior change.

1. Danger source-governance warnings on unsupported (declined) routes are no
   longer counted as failures. The app attaches a weak-evidence danger warning
   when relevance verdict is "none", which on a refusal is the correct reader
   signal ("no source-backed evidence, do not act"), not a governance failure.
   The metric now counts danger warnings only on answered routes (a delivered
   answer standing on dangerous sourcing). For genuinely out-of-scope cases the
   warning's presence is positively asserted so a silent regression is caught;
   for supported questions that were wrongly refused, the routing failure is
   reported elsewhere and no misleading missing-warning noise is added.
   Verified on the live release eval: source_governance_danger_failure_rate
   0.225 -> 0, all 9 affected cases are correctly-declined out-of-scope queries.

2. Re-accept the bounded source-metadata debt at a 0.6 review-required ceiling
   (expiry unchanged, 2026-07-31). The rate rose 0.14 -> 0.5398 with no corpus
   change: the relevance-first ranking work (PR #118 / PR #130) removed
   governance metadata weighting from selection ordering, so review-flagged
   sources are no longer buried and the metric now reports true corpus state.
   docs/source-review-priority-2026-07-02.md lists the 25 documents occupying
   golden top-5 slots, highest-frequency first; reviewing the top 12 returns the
   rate to ~0.12, after which the ceiling should be tightened back toward 0.2.

Note: the eval still reports separate blocking failures (grounded_supported,
one prompt-injection false-positive, citation, extractive latency) that are
pre-existing answer-quality regressions unrelated to these two changes and
tracked separately.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(eval): key danger-warning failures on grounded, not route (review fixes)

Addresses two automated review notes on the danger-warning reclassification:

- Exempt non-grounded refusals however they arise. finalizeRagAnswerQualityCore
  can convert a fast/strong/extractive answer into an evidence-gap refusal by
  setting grounded=false while preserving the original routingMode, so a
  route-based check could still fail the gate on a declined answer. Key the
  danger-warning failure on grounded === true (an answer was actually
  delivered) instead of route !== "unsupported".

- Drop the toothless positive "missing danger warning" assertion. It only
  surfaced in failed_cases and never fed a thresholded metric, so it could not
  gate; making it gate would introduce new blocking failures on the current
  corpus (e.g. close-title-noise), which we explicitly must not do. The
  grounded-based check is the clean, robust core.

Behavior unchanged on the current corpus: danger rate stays 0 (all danger
warnings are on grounded=false refusals). Unit tests updated to cover the
finalize-core evidence-gap refusal path. vitest 8/8, typecheck + lint clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
BigSimmo added a commit that referenced this pull request Jul 14, 2026
Audit 2026-07-13 deferred items D4 + D5, completed as zero-behavior-change
plumbing per the #118 lesson (no governance weighting without golden-eval
proof) — the levers ship OFF and are activated by the operator only behind
a green golden eval once OpenAI quota recovers.

- D4: secondStage.unknownCurrentnessPenalty in ranking-config (default 0,
  clamped non-negative) applied in the second-stage rerank when
  document_status is 'unknown'. Activation = RAG_RANKING_CONFIG JSON
  override, the module's designed eval-gated tuning path.
- D5: deriveTrust broadening behind NEXT_PUBLIC_RAG_TRUST_CAP_ALL_CLAIMS
  (unset = false): when on, unverified-authority evidence caps trust to
  medium for ALL supported claims, not just high-risk ones. NEXT_PUBLIC
  because answer-render-policy renders client-side.
- Tests lock both the zero-change defaults and the flag-on behavior.

Verified: verify:cheap green (2,310 tests), eval:rag:offline green.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
BigSimmo pushed a commit that referenced this pull request Jul 30, 2026
BigSimmo pushed a commit that referenced this pull request Jul 30, 2026
BigSimmo pushed a commit that referenced this pull request Jul 30, 2026
…able

Every open PR sampled carries the same ci/circleci: verify failure, including
#1400 which is a docs-only AGENTS.md change, so it is not branch-specific. The
job's whole contents were mirrored locally on #1396's tip and all of it is
green, including the PyMuPDF-gated PDF tests under a venv built exactly as
.circleci/config.yml builds it. That puts the fault in the job environment.

Recorded with the quota hypothesis marked explicitly unverified: the CircleCI
project is private, no session has a CircleCI token, and the unauthenticated
API returns "Build not found". Reading the failing step needs an operator.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01361jh3eYVjJCzXWjAhdZiF
BigSimmo added a commit that referenced this pull request Jul 30, 2026
… PR babysitting (#1421)

* issues: capture two CI/merge operational findings from PR babysitting

#117: this repo's CI (on: pull_request with default types) doesn't
retrigger on the draft-to-ready transition, only on
opened/synchronize/reopened — a marked-ready PR can sit with a
minimal check set until an actual new commit lands. #118: GitHub's
update-branch API doesn't honor the merge=union .gitattributes driver
on docs/branch-review-ledger.md, so it can 422 with a false conflict
that a local git merge resolves cleanly. Both observed today on PR
#1406.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Re9ERtQwJ82ErbAnahAhsa

* Address Codex review: clarify ledger rows #125/#126 wording

#125: spell out that ready_for_review must be added alongside the
existing pull_request.types defaults, not replace them.
#126: qualify the update-branch fallback push with the same
explicit user confirmation the provider boundary already requires.

* docs(issues): capture #1396's unaddressed physical-device chrome gate

PR #1396 repeatedly declared physical-device Safari/PWA acceptance
(docs/phone-chrome-physical-acceptance.md) as required before merge
because headless Chromium cannot certify Safari chrome minimisation
or cold-launch PWA paint, then merged with the checklist still blank.
Also notes a related missing pre-paint/cold-load hydration test the
same PR's review flagged but never filed.

* style: fix table padding drift from the main merge

npx prettier --write after merging main (9e2fe44) — a table cell
width shifted during the merge and format:check would have caught it.

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: BigSimmo <BigSimmo@users.noreply.github.com>
BigSimmo pushed a commit that referenced this pull request Jul 30, 2026
Fourth genuine conflict on docs/outstanding-issues.md, confirmed with
git merge-tree before acting. Main's #1470 rewrote #13 as
measurement-gated while this branch changed #17, #105 and added #147.
Verified by hash that main touched neither #17 nor #105 — both are
byte-identical at the merge base and at origin/main — so those rows take
this branch's version and #13 takes main's, rather than either side
winning wholesale.

Also de-duplicated against main's new #118, which already owns adopting
the Lighthouse baseline and flipping enforce. #147's gate paragraph now
cross-references it instead of restating it, and contributes the
constraint #118 was missing: pinning a baseline today would bake CLS
0.363 on /dsm in as the accepted value, so the CLS fixes must land first
or the gate ratifies the breach.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01361jh3eYVjJCzXWjAhdZiF
BigSimmo pushed a commit that referenced this pull request Jul 31, 2026
Resolves the fifth conflict on docs/outstanding-issues.md by taking
main's side on both hunks, after verifying that was correct rather than
convenient: main renumbered the queue to fix a duplicate rank 18 this
branch introduced, and it archived #138 and #146 rather than dropping
them (confirmed present in the archive table). The resulting id set is
byte-identical to main's.

#150 escalated. That row records CodeRabbit reviewing none of a full
day's PRs and rests explicitly on the Codex connector being the
surviving reviewer that "found three real defects that had survived
local gates". On PR #1505 Codex posted its own usage-limit notice
alongside CodeRabbit's spending-cap notice, so that PR received zero
automated review and so will anything opened while both caps hold. This
is the same issue with its fallback removed, not a new one — which
matters because the single Codex finding on #1459 was correct and
caught a verification that had matched the wrong component, closing
#105 on bad evidence. Local gates did not catch it.

The attribution harness lands as scripts/measure-cls-attribution.mjs.
#147's next step needs a before/after CLS pair, and #118 will want
element attribution again; without this, both mean re-deriving it. Build
and serve mirror run-lighthouse-budget.mjs so the numbers sit beside the
Lighthouse reports. Two traps that cost real time are encoded rather
than left to be rediscovered: CHROME_PATH must be set where the browser
is outside a standard location, and an init script attaching a
MutationObserver to document.documentElement before <html> exists throws
and silently takes the CLS observer with it — so the script now fails
loudly on an all-zero result instead of reporting a false clean bill.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01361jh3eYVjJCzXWjAhdZiF
BigSimmo added a commit that referenced this pull request Aug 4, 2026
These came out of the DS V2 wave with no ledger row and would otherwise have
evaporated with the session: the stale PR-J governance review record, the
COMPONENTS.md section 0 drift plus the undocumented optionality-marker
contract change, the undocumented answer-copy-payload module, the missing
ADOPTION.md section 7 proof shots, and the two orphan builder branches and
leftover dev servers. The sixth residual, visual baselines staying zero and
advisory, is already ledger #118 and was deliberately not duplicated.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
cursor Bot pushed a commit that referenced this pull request Aug 4, 2026
…o states that need one (#1612)

* docs(issues): record #230 live answer generation timeout

Production triage 2026-08-04 measured the live failure: OpenAI generation
overruns OPENAI_ANSWER_TIMEOUT_MS and the answer degrades to source-only,
while retrieval passes its gate with 12 candidates across 5 documents. The
2026-08-04 handover's four ranked hypotheses (query embedding, Supabase RPC,
source governance, corpus) are all ruled out by that evidence.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* feat(ds): let EmptyState carry a heading, and adopt the two states that need one

EmptyState rendered its title as a <p>, so any empty state that owns a region
of the page could not adopt it without losing its heading. /dsm/search lost
exactly that when it adopted the primitive (#224), and the main document-search
empty state stayed hand-rolled because ui-smoke pins a heading role on it
(#217).

Add an opt-in headingLevel prop. It is deliberately un-defaulted: most adopted
call sites sit inside a card that already owns its region's heading, and
promoting every title would inject an outline level the page never declared.

- /dsm/search passes headingLevel={2}, restoring the heading it had before.
- The document-search empty state adopts EmptyState with headingLevel={3},
  which also converts it from a bespoke centred card to the shared state
  surface and gives it the polite live region every other adopted empty state
  has.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(issues): correct #230 — the fast route budget binds, not the env timeout

The first row named OPENAI_ANSWER_TIMEOUT_MS as the constraint. It is not,
and production does not set it. The query routes fast, so answerRouteBudgetMs.fast
(25s) is the ceiling and generationRequestTimeoutMs holds back the 2s recovery
reserve on top; the measured 24.7s is that budget being spent.

Also records the half with clinical consequence: answerRouteResultCanBeCached
excludes a route-deadline-exceeded answer but not one degraded by the OpenAI
request timing out, so a generation_fallback answer can be cached and re-served
after the provider recovers.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(ledger): record the PR #1612 review

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(issues): capture the five Wave 5 close-out residuals (#231-#235)

These came out of the DS V2 wave with no ledger row and would otherwise have
evaporated with the session: the stale PR-J governance review record, the
COMPONENTS.md section 0 drift plus the undocumented optionality-marker
contract change, the undocumented answer-copy-payload module, the missing
ADOPTION.md section 7 proof shots, and the two orphan builder branches and
leftover dev servers. The sixth residual, visual baselines staying zero and
advisory, is already ledger #118 and was deliberately not duplicated.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(issues): archive #217 and #224 completed by EmptyState headingLevel

PR #1612 added the opt-in headingLevel prop and adopted the DSM and
document-search empty states that needed it. Close both ledger rows so
/issues stops treating finished work as outstanding.

* docs(issues): correct #231 summary to name the binding fast-route budget

The detail already established that OPENAI_ANSWER_TIMEOUT_MS is not the
binding ceiling — answerRouteBudgetMs.fast = 25000 is. Align the summary
so triage does not raise a non-binding setting, and backtick the
src/lib/rag/** path so markdownlint MD037 stays quiet.

* ci: retrigger Production UI after known Services viewport-shrink flake

Production UI (1) failed once on ui-phone-scroll-page-owned:577
(Services results re-settle after viewport shrink) — same signature as
the 2026-07-31 flake noted in the branch-review ledger. Unrelated to
EmptyState headingLevel; shards 2/3 and critical passed. Cannot
rerun-failed via this token (403), so retrigger the pull_request run.

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
BigSimmo added a commit that referenced this pull request Aug 8, 2026
Same defect Codex flagged on PR #1719: the commits recording these measurements
were authored 2026-08-08 (07:xx UTC), so a 2026-08-09 stamp places every
re-measurement after the commit that recorded it. Four occurrences, in the #118,
#269 and #270 rows.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
BigSimmo added a commit that referenced this pull request Aug 8, 2026
cursor Bot pushed a commit that referenced this pull request Aug 8, 2026
Codex P2 on PR #1724: the red advisory is a stale Chromium 150 baseline
overwrite from #1710 against a job already pinned to Playwright Chromium
151 (#1716), not a fresh CI-runner 150→151 bump. Prefer restoring or
comparing the known 151 baseline over a blind --update.

Co-authored-by: BigSimmo <BigSimmo@users.noreply.github.com>
cursor Bot pushed a commit that referenced this pull request Aug 8, 2026
Prefer main's outstanding-issues shared queue (#277 / next-id=278 from
#1725), then re-apply this PR's #276 archive and corrected #118 Lighthouse
diagnosis.

Co-authored-by: BigSimmo <BigSimmo@users.noreply.github.com>
BigSimmo added a commit that referenced this pull request Aug 8, 2026
The first commit adopted the six PNGs but missed `tests/design-system-adoption.test.ts`,
a committed contract that governs exactly this adoption and that no handover in this
series mentions. It was dormant only because it had no committed baselines to check.
`npm run verify:cheap` caught it: 9 failed / 5586 passed.

Three things it requires, all now met:

- `tests/__screenshots__/linux/provenance.json` (schemaVersion 2) binding the goldens
  to their capture: platform `linux`, runner `ubuntu-24.04`, hosted-CI artifact
  `visual-baseline-31251091603`, candidateSourceHead `bc33d414e`, and a SHA-256 plus
  pixel dimensions per candidate. The reviewer attestation is a human one — recorded
  after the six PNGs were actually looked at, not asserted on their behalf.
- `AWAITING_BASELINE` must stay a *static literal* `new Set([...])`, and the suite may
  differ from the capture commit in nothing but those values. The explanatory comment
  added in the first commit violated that, and `new Set<string>()` did not parse as a
  literal set at all. Both reverted; the suite diff is now exactly the six ids.
- Only an allowlisted set of paths may change alongside the adoption, which is why the
  `tests/__screenshots__/README.md` note from the first commit is reverted here. The
  document-viewer limitation it recorded is in the PR description instead.

The fixture change is the one substantive fix. `initialiseCandidateRepository` seeded
its candidate-source commit from the *live* `tests/ui-visual-baseline.spec.ts`, so every
fixture silently depended on the repository not having adopted its baselines yet — and
the commit that empties `AWAITING_BASELINE`, the outcome the contract exists to permit,
made the fixture's own candidate head fail "must contain exactly the canonical six ids".
It now defaults to the explicit canonical list. A fixture states its precondition rather
than borrowing it from the tree under test.

Verified: tests/design-system-adoption.test.ts 51 passed (51). Mutation-tested by
corrupting a provenance SHA-256, which fails with the expected mismatch.

Refs #118
cursor Bot pushed a commit that referenced this pull request Aug 8, 2026
The superseded banner said all six visual baselines were adopted under
#118, but the ledger still has #118 open and tests/__screenshots__/
holds only README.md. Keep the accurate Ubuntu-CI generation point
without implying visual-regression protection exists yet.

Co-authored-by: BigSimmo <BigSimmo@users.noreply.github.com>
BigSimmo added a commit that referenced this pull request Aug 8, 2026
* docs(design-system): stamp the 7 August handover superseded

Nine open ledger rows (#261, #262, #264-#270) cite
docs/design-system/HANDOVER-2026-08-07.md as their Source, and
docs/design-system/README.md sent anyone picking the work up cold straight to it as
"measured state, the ordered plan". Four of its figures have since been disproved,
and the corrections were written into the ledger rows rather than the document, so
the document still asserts the originals. A session scoping from it re-derives work
that is already known wrong. Recorded as #277.

A banner rather than a rewrite, which is what #277 asks for: the corrections already
live in the rows, and duplicating them re-creates the drift this fixes. The banner
names the four measured errors -- the "229 --shadow-tight aliases" that is really a
seven-token total mislabelled as one token (100 sites across 55 files, 228 total),
the 24-vs-23 unadopted count, the "baselines cannot be generated on Windows"
conclusion that overlooked the ubuntu CI job now used under #118, and #270's 22-site
premise that has zero same-variant pairs -- then points at the rows.

The README now sends a cold start to docs/outstanding-issues.md and marks the
handover superseded in the link text itself, where the misdirection was.

Deliberately not done: the document is not deleted, moved, or corrected in place.
The nine Source citations, the PR and commit record, and its verification and gotcha
sections are provenance the ledger is meant to preserve, and silently correcting it
would leave those rows citing a document that no longer says what they were derived
from.

Verified: npm run docs:check-links -- "docs link check passed: 1667 repo path
references resolve"; npm run docs:check-inventory -- current; npm run format:check --
"All matched files use Prettier code style!".

Refs #277

* docs(design-system): drop false #118 baseline-adoption claim

The superseded banner said all six visual baselines were adopted under
#118, but the ledger still has #118 open and tests/__screenshots__/
holds only README.md. Keep the accurate Ubuntu-CI generation point
without implying visual-regression protection exists yet.

Co-authored-by: BigSimmo <BigSimmo@users.noreply.github.com>

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: BigSimmo <BigSimmo@users.noreply.github.com>
BigSimmo added a commit that referenced this pull request Aug 8, 2026
…provenance contract (#1729)

* test(visual): adopt the six linux visual baselines from CI run 31251091603

`tests/__screenshots__/` has held only a README since the visual gate landed, so
every target sat in `AWAITING_BASELINE`, skipped rather than compared, and no PR
could prove it had not moved the look. That is the blocker `#118` records and the
one `#269` is formally waiting on.

Adopted from the ubuntu CI artifact `visual-baseline-31251091603` — a `push` on
`main` at `bc33d414e`, never from a developer machine: `snapshotPathTemplate` is
platform-scoped, so win32 PNGs land in `win32/` and the ubuntu job never reads
them, and font hinting alone would make every later run red.

All six goldens were reviewed before committing, and stability was measured rather
than assumed: five of the six are byte-identical by SHA-256 to the candidates from
the earlier independent run 31249978408 at `2069b1f5c`. Only `document-viewer`
differs, and those two runs straddle #1705, which changed `DocumentViewer.tsx`.

`AWAITING_BASELINE` is now empty, so a missing golden is once again unambiguously a
fault rather than a declared exemption. The two honesty tests keep that list from
rotting in either direction.

Deliberately NOT included: promotion. `visual-baseline` stays `continue-on-error`
and outside `pr-required`. The workflow comment asks for a soak first, and adding
the job to `pr-required` and dropping `continue-on-error` belong in one later edit.

Known limitation recorded in the README rather than papered over: the
`document-viewer` target clips a ~2900px `#main-content` against a 900px viewport
and contains viewport-pinned chrome (`sm:sticky` header, `sm:fixed` composer),
which composites partway down the stitched capture and overlaps content. It is
deterministic, so the comparison still means something, but it predates #1705 and
would be quieter with a narrower clip or a mask.

Refs #118

* test(visual): satisfy the committed baseline provenance contract

The first commit adopted the six PNGs but missed `tests/design-system-adoption.test.ts`,
a committed contract that governs exactly this adoption and that no handover in this
series mentions. It was dormant only because it had no committed baselines to check.
`npm run verify:cheap` caught it: 9 failed / 5586 passed.

Three things it requires, all now met:

- `tests/__screenshots__/linux/provenance.json` (schemaVersion 2) binding the goldens
  to their capture: platform `linux`, runner `ubuntu-24.04`, hosted-CI artifact
  `visual-baseline-31251091603`, candidateSourceHead `bc33d414e`, and a SHA-256 plus
  pixel dimensions per candidate. The reviewer attestation is a human one — recorded
  after the six PNGs were actually looked at, not asserted on their behalf.
- `AWAITING_BASELINE` must stay a *static literal* `new Set([...])`, and the suite may
  differ from the capture commit in nothing but those values. The explanatory comment
  added in the first commit violated that, and `new Set<string>()` did not parse as a
  literal set at all. Both reverted; the suite diff is now exactly the six ids.
- Only an allowlisted set of paths may change alongside the adoption, which is why the
  `tests/__screenshots__/README.md` note from the first commit is reverted here. The
  document-viewer limitation it recorded is in the PR description instead.

The fixture change is the one substantive fix. `initialiseCandidateRepository` seeded
its candidate-source commit from the *live* `tests/ui-visual-baseline.spec.ts`, so every
fixture silently depended on the repository not having adopted its baselines yet — and
the commit that empties `AWAITING_BASELINE`, the outcome the contract exists to permit,
made the fixture's own candidate head fail "must contain exactly the canonical six ids".
It now defaults to the explicit canonical list. A fixture states its precondition rather
than borrowing it from the tree under test.

Verified: tests/design-system-adoption.test.ts 51 passed (51). Mutation-tested by
corrupting a provenance SHA-256, which fails with the expected mismatch.

Refs #118
BigSimmo added a commit that referenced this pull request Aug 8, 2026
… not only the first adoption (#1742)

`validateCandidateSourceBinding` required `candidateSourceHead` to name a commit
whose `AWAITING_BASELINE` still declared the canonical six, and the current tree to
declare none. That models the FIRST adoption and nothing else. After #1729 landed,
no commit on main declares the six again, so no future capture head could ever
satisfy it -- the binding was satisfiable exactly once, and that once has passed.

The consequence is only latent today, and this commit does not pretend otherwise:
`validateLinuxVisualBaselineSet` short-circuits on `declaredPaths.length === 0` and
no surface yet declares `baseline.status: "committed"`, so the whole contract is
dormant and refreshing a golden is currently ungated. The defect bites the moment
those surfaces are declared (#242) -- at which point the first intentional design
change would leave the goldens red with no supported way to re-adopt them, which is
precisely backwards for a gate meant to make design changes provable.

So the binding now accepts either shape:

  FIRST ADOPTION  candidate head declares the six, current declares none.
  REFRESH         both ends empty, suite byte-identical.

A refresh is the ordinary case once baselines exist: a surface is deliberately
re-shot and its goldens replaced while the suite itself does not change at all.
Normalisation already blanks the AWAITING values on both sides, so the
"suite changed beyond the AWAITING declaration" check stays exact for both.

Nothing is loosened beyond that. A candidate head with five ids, seven ids, a
dynamic set, a spread or a duplicate still fails, as do a non-empty current list, a
non-ancestor head, a hash mismatch, a wrong platform and an unreviewed set. The one
previously-untested shape -- a refresh -- now has its own case, because the gap was
invisible precisely because nothing exercised it.

Verified: tests/design-system-adoption.test.ts 51 passed (51), including the new
refresh case; check:design-system-adoption and check:design-system-contract exit 0;
format:check clean.

Refs #242, #118
cursor Bot pushed a commit that referenced this pull request Aug 8, 2026
…seline, and add a baseline adopt helper (#1743)

* feat(design-system): mask the pinned chrome in the document-viewer baseline, and add a baseline adopt helper

Two changes that make refreshing a golden after a deliberate design change cheap
and readable, which is what the pixel gate needs in order to be read at all.

Mask the viewport-pinned chrome (#278). The document-viewer target clips a ~2900px
region against a 900px viewport, and it contains a `sm:sticky sm:top-0` header and
a `sm:fixed` composer. Playwright stitches an oversized element capture, so both
land partway DOWN the image, overlap whatever content sits behind them at that
offset, and move whenever content above them changes height -- so an unrelated edit
anywhere on the page redrew two bands of the golden and inflated every diff. Masked
rather than clipped away, because both are real chrome that belongs in the frame
and narrowing the selector would drop the rail panels this target exists to watch.
Their own geometry is covered by the phone-chrome contracts in
docs/search-chrome-behaviour.md, not by this pixel gate.

Fail loudly when a mask matches nothing. A mask selector that matches no element
masks nothing, silently -- the golden keeps comparing the region the mask was meant
to exclude while the declaration reads as protection that is not there. Renaming a
class is enough to cause it. Every declared mask must now resolve to at least one
element before the comparison is trusted. That guard is what makes the mask above
safe to rely on rather than merely present.

Add scripts/adopt-visual-baselines.mjs (npm run design-system:baselines:adopt).
Adopting previously meant hand-copying six PNGs and hand-assembling provenance.json
with a SHA-256 and pixel dimensions per candidate, the capture commit, the run id
and the reviewer attestation. Doing that by hand on every design change is the
friction that makes people skip the refresh and leave a red advisory standing.

It resolves candidates from both shapes the artifact can take -- visual-candidates/
when a target was awaiting a baseline, and <id>-actual.png when it compared and
differed. It refuses a missing or non-numeric run id, a short or unknown capture
commit, and a missing --reviewed-by, because that field records a HUMAN review of
the images. It never captures screenshots: baselines are platform-scoped and a
developer-machine shot lands where ubuntu CI never reads it. Dry run by default.

The document-viewer golden is NOT refreshed here -- masking changes its pixels, so
it must be re-shot from a CI run that already contains this mask. That is the
refresh loop the helper exists for, and it needs this to land first.

Verified: tsc 0 errors; lint exit 0; check:design-system-adoption exit 0;
docs:check-inventory current at 232 npm scripts; docs:check-links exit 0;
format:check clean. Helper exercised against two real artifacts -- it resolves all
six candidates from run 31251091603 and correctly refuses run 31254917796, which
was all-green and so contains no candidate images -- and all three input guards
were confirmed to reject. Dry run wrote nothing.

Refs #278, #118

* fix(design-system): support partial visual baseline refresh adoption

When CI changes only some surfaces, passing targets emit no *-actual.png
and unchanged images live under tests/__screenshots__/ in the artifact.
The adopt helper now resolves candidates from diff output, artifact
baselines, or committed goldens, records refresh provenance (including
replacedCandidateIds), and validates capture-head AWAITING_BASELINE
binding before writing. Also fix git cat-file reachability checks and
run mask-selector validation before candidate capture.

* chore(design-system): refresh document-viewer baseline after chrome masks

Adopt the masked document-viewer capture from visual-baseline-31267928439
(tip ac1d0dd) via the partial-refresh helper. Clears the expected advisory
diff introduced when the sticky header and fixed composer were masked.

Co-authored-by: BigSimmo <BigSimmo@users.noreply.github.com>

* fix(design-system): gate retained baselines on junit and uniquify the mask

Copilot: retaining screenshots from the artifact without visual-junit proof
could bless a stale golden for a target that failed before emitting an actual.
Require a passing junit case for every retained id, refuse all-green refreshes,
and only treat *-actual.png as a fresh diff. Also mask
[data-document-sticky-header] instead of .edge-glass-header so the fail-loud
guard cannot pass against the universal search header after a DocumentViewer rename.

Co-authored-by: BigSimmo <BigSimmo@users.noreply.github.com>

* chore(design-system): refresh masked document-viewer baseline after main sync

Re-adopt from visual-baseline-31268982766 on the post-#1741 merge tip so the
golden includes the unique data-document-sticky-header mask and the viewer
rework from main.

Co-authored-by: BigSimmo <BigSimmo@users.noreply.github.com>

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: BigSimmo <BigSimmo@users.noreply.github.com>
cursor Bot pushed a commit that referenced this pull request Aug 8, 2026
)

* ci(visual): stop running the visual baseline job on pull requests

Owner decision, 2026-08-09: the pixel gate taxes exactly the work it exists to
support. During a redesign the honest result is red on every UI pull request, and
the baselines cannot be refreshed until a run has produced the new images, so that
red is unavoidable rather than actionable.

It was never merge-blocking -- `continue-on-error: true`, and absent from
`pr-required`, whose needs list is [changes, static-pr, safety, coverage, build,
container-images, ui-critical-fast, ui-critical, db-reset-verify]. But a red check
nobody can act on mid-iteration is friction whether or not it gates, and it
generated a steady stream of failure notifications on PR #1743 while the only
correct response was "yes, that is the change".

Drift is still caught. The job keeps running on pushes to main and release
branches, on the weekly schedule, and on demand via workflow_dispatch. That moves
the signal to where a refresh is cheap -- after the change has landed, against a
run whose artifact already contains the new images -- instead of interrupting the
change. One command refreshes it:
npm run design-system:baselines:adopt -- --from <artifact> --run-id <id> --head <sha> --reviewed-by "<name>" --write

Nothing else changes: the job, its targets, its provenance contract and its
advisory status are untouched, and no required check is added or removed.

Verified: npm run check:github-actions passed; ci-cache-safety 18 passed, 11
skipped; no test pins this job to pull_request; format clean.

Refs #118

* issues: record the owner decision that visual-baseline stays off pull requests

* fix(ci): keep visual-baseline off merge_group as well as PRs

Allowlist push/schedule/workflow_dispatch so merge-queue UI churn does not
re-run the advisory pixel gate, and document the schedule in testing.md.

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: BigSimmo <BigSimmo@users.noreply.github.com>
BigSimmo pushed a commit that referenced this pull request Aug 12, 2026
…re-scope the wave plan

Archived with source evidence:
- #166 the verify-before-use caveat ships (verification-notice.tsx:52, rendered
  unconditionally by AnswerCard at answer-card.tsx:108). I reported this row as
  open twice before reading the code; the archive says so.
- #208 buildAnswerClipboardText composes renderCopyText rather than replacing
  formatAnswerRenderCopyText (answer-copy-payload.ts:121-138).
- #216 AnswerCard is adopted on the answer surface and the manifest agrees.

Re-scoped #250: its wave plan names A1 rows that are now closed and hygiene
rows that are in flight, so as written it misdirects. Recorded what actually
remains (#147, #117, #118, #98, #189).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017paT42ZVMf8jaLtkjFxdy5
BigSimmo pushed a commit that referenced this pull request Aug 12, 2026
Measured, not assumed. Two independent runs of the offline harness this row
specifies, on Chromium 141 — the same browser that produced the row's original
figures, so before/after is like-for-like. Mobile CLS (run1/run2 vs recorded):

  /dsm               0.035 / 0.035   was 0.363
  /documents/search  0.000 / 0.000   was 0.220
  /                  0.013 / 0.013   was 0.023
  /forms             0.081 / 0.081   was 0.310 local
  /therapy-compass   0.000 / 0.000   was 0.240 local

The 128px reserve round trip is gone. The cause was fixed by PR #1616, not by
this session: use-phone-overlay-chrome-reserve.ts keeps the CSS seed until an
observer candidate holds across an 80ms quiet window and publishes only via
ResizeObserver — both remedies the row proposed — and its comment at :61 cites
the row by number.

Two controls make this a pass rather than the false clean the row warns about:
values are not uniformly 0.000, so the observer is live; and the /dsm variance
the row recorded (0.363 vs 0.219) has itself vanished, which is what you expect
once the transient write causing it stopped.

Also recorded: #118 updated — the Lighthouse browser drift is 141-vs-151, wider
than the 150-vs-151 the row states, so the budget cannot grade from this
container at all. New row #308 for desktop /documents/search CLS 0.119, stable
against a 0.119 baseline, above threshold, and outside #147's mobile scope —
which also makes the row's "desktop passes everywhere 0.016-0.097" stale.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017paT42ZVMf8jaLtkjFxdy5
BigSimmo added a commit that referenced this pull request Aug 17, 2026
* docs(issues): queue inbox request for mobile CLS regression on /

PR #2050's Lighthouse budget check surfaced a real, reproducible mobile-root
CLS regression (0.236 vs the committed 0.013 baseline) unrelated to that
PR's diff. Cross-references the previously-resolved #147/#118 CLS work so
it isn't lost; reconciliation into the canonical ledger happens separately.

* docs(ledger): record review for PR #2059 (CLS regression issue log)

---------

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant